Slot migration improvement #445
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ding process, specifically during slot migrations in the face of primary failures. Signed-off-by: Ping Xie <pingxie@google.com>
Signed-off-by: Ping Xie <pingxie@google.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## unstable #445 +/- ##
============================================
+ Coverage 68.43% 68.89% +0.45%
============================================
Files 109 109
Lines 61681 61785 +104
============================================
+ Hits 42214 42565 +351
+ Misses 19467 19220 -247
|
madolson
approved these changes
May 7, 2024
|
i see the (squash merge) commit message is missing (Signed-off-by footer is also missing) |
zuiderkwast
added a commit
that referenced
this pull request
May 24, 2024
After READONLY, make a cluster replica behave as its primary regarding returning ASK redirects and TRYAGAIN. Without this patch, a client reading from a replica cannot tell if a key doesn't exist or if it has already been migrated to another shard as part of an ongoing slot migration. Therefore, without an ASK redirect in this situation, offloading reads to cluster replicas wasn't reliable. Note: The target of a redirect is always a primary. If a client wants to continue reading from a replica after following a redirect, it needs to figure out the replicas of that new primary using CLUSTER SHARDS or similar. This is related to #21 and has been made possible by the introduction of Replication of Slot Migration States in #445. ---- Release notes: During cluster slot migration, replicas are able to return -ASK redirects and -TRYAGAIN. --------- Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
This was referenced Jun 2, 2024
PingXie
added a commit
that referenced
this pull request
Jul 16, 2024
Fixes a regression introduced in PR #445, which allowed a message from a replica to update the slot ownership of its primary. The regression results in a `replicaof` cycle, causing server crashes due to the cycle detection assert. The fix restores the previous behavior where only primary senders can trigger `clusterUpdateSlotsConfigWith`. Additional changes: * Handling of primaries without slots is obsoleted by new handling of when a sender that was a replica announces that it is now a primary. * Replication loop detection code is unchanged but shifted downwards. * Some variables are renamed for better readability and some are introduced to avoid repeated memcmp() calls. Fixes #753. --------- Signed-off-by: Ping Xie <pingxie@google.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR significantly enhances the reliability and automation of
the Valkey cluster re-sharding process, specifically during slot
migrations in the face of primary failures. These updates address
critical failure issues that previously required extensive manual
intervention and could lead to data loss or inconsistent cluster states.
Enhancements
Automatic Failover Support in Empty Shards
The cluster now supports automatic failover in shards that do not
own any slots, which is common during scaling operations. This
improvement ensures high availability and resilience from the
outset of shard expansion.
Replication of Slot Migration States
All
CLUSTER SETSLOTcommands are now initially executed on replicanodes before the primary. This ensures that the slot migration state
is consistent within the shard, preventing state loss in the event of
primary failure. A new timeout parameter has been introduced, allowing
users to specify the duration in milliseconds to wait for replication
to complete, with a default set at 2 seconds.
Recovery of Logical Migration Links
The update automatically repairs the logical links between source
and target nodes during failovers. This ensures that requests are
correctly redirected to the new primary in the target shard after
a primary failure, maintaining cluster integrity.
Enhanced Support for New Replicas
New replicas added to shards involved in slot migrations will now
automatically inherit the slot's migration state as part of their
initialization. This ensures that new replicas are immediately
consistent with the rest of the shard.
Improved Logging for Slot Migrations
Additional logging has been implemented to provide operators with
clearer insights into the slot migration processes and automatic
recovery actions, aiding in monitoring and troubleshooting.
Additional Changes
cluster-allow-replica-migration
When
cluster-allow-replica-migrationis disabled, primary nodes thatlose their last slot to another shard will no longer automatically become
replicas of the receiving shard. Instead, they will remain in their own
shards, which will now be empty, having no slots assigned to them.
Fixes #21.